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Advanced  Visualization  for  Operational  Assessment 


Combined  Air  and  Space  Operations  Centers  (CAOC)  have  traditionally  relied  on  combat 
assessment  to  evaluate  operational  effectiveness.  However,  a  more  flexible  model  of 
operational  assessment  (OA)  is  required  to  achieve  continued  maturation  of  an  Effects- 
based  Operations  (EBO)  approach.  Such  a  model  will  account  for  non-military 
considerations:  Political,  economic,  social  and  infrastructure  dimensions.  It  also  will 
allow  assessors  to  consider  asymmetric  factors  such  as  counter-insurgency  and  terrorism 
control  in  assessing  effectiveness.  Unfortunately,  OA  currently  receives  relatively  little 
attention,  and  there  are  few  tools  to  support  OA  focused  on  effects  rather  than  attrition. 
We  carried  out  an  analysis  of  Effects-based  Assessment  (EBA),  focusing  on  information 
requirements  and  data  needs,  decisions  and  their  critical  cues,  and  common  errors.  We 
then  developed  a  model  of  the  work  management,  organization,  products  and  cognition  of 
EBA.  In  this  paper  we  will  describe  our  cognitive  systems  engineering  methodology 
used  to  analyze  EBA  and  the  findings  that  constitute  our  understanding  of  the  process. 

We  then  will  offer  a  visualization  system  designed  to  enhance  the  OA  team’s 
understanding  of  operations  as  they  relate  to  the  complexity  of  effects  in  EBO,  that  is,  to 
desired  and  undesired  effects,  nth-order  effects  and  kinetic  versus  non-kinetic  effects. 
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Introduction 

The  Air  Force  is  in  the  midst  of  a  technology  transition  that  will  enable  and 
support  operators  in  their  decision  making  strategies  by  enabling  them  to  have  access  to 
.  the  information  needed,  in  a  timely  manner,  to  maintain  situation  awareness  over  the 
battlespace,  and  to  work  jointly  with  other  commands.  The  overarching  goal  of  this 
transition  is  to  provide  a  network-centric  warfare  (NCW)  capability  that  in  military 
operations  theory  holds  that  the  seamless  networking  of  friendly  force  elements  will 
bring  about  an  increase  in  combat  power  (Alberts,  Gartstka,  &  Stein,  1999).  The  basic 
idea  is  that  related  mission  applications  should  be  integrated  into  a  single  managed  "C2 
Node". 


Included  in  this  technology  transition  is  the  advancement  and  implementation  of 
an  Effects  Based  Operation  (EBO)  methodology.  The  goal  for  the  Air  Force  is  to  enforce 
a  standardization  of  this  methodology  both  in  practice  and  in  system  development.  This 
enforcement  is  occurring  because  operations  have  become  complex  and  more  difficult  to 
keep  track  of,  especially  when  joint  forces  are  employed. 

Up  to  this  point,  systems  have  been  designed  to  support  a  kinetic  (Strategy-to- 
Task),  attrition-based  approach  to  war.  This  approach  extends  to  both  strategy  planning 
and  assessment.  This  approach  focuses  the  Strategy  Planning  Division  primarily  on 
tactical  actions  and  a  localized  evaluation  of  mission  outcomes  in  the  form  of  battle 
damage  assessment  (BDA).  Attrition-oriented  approaches,  focused  solely  on  kinetic 
actions  and  localized  results,  prevents  consideration  of  other  important  factors  in  modem 
military  operations.  These  include  economic,  social,  political  and 
information/infrastructure  factors  that  can  substantially  affect  the  end-game  of  an 
operation.  Attrition-based  approaches  also  do  not  allow  for  planning  and/or  re-planning 
based  on  an  in-depth  understanding  of  the  effects  that  actions  have  produced.  Lastly,  it 
does  not  support  an  understanding  of  the  duration  of  an  effect,  nth  order  effects,  and 
unintended  effects. 

In  order  to  transition  successfully  from  an  attrition-based,  strategy-to-task 
approach,  the  technology  created  to  support  the  warfighter  must  be  designed  to  better  aid 
them  in  addressing  the  issues  cited  above:  Relating  actions  to  intended  effects; 
performing  predictive  assessment  in  addition  to  assessment  during  and  after  execution; 
carrying  out  causal  link  analysis;  and  understanding  nth-order  effects  within  the  context 
of  economic,  social,  and  political  dimensions  (in  addition  to  traditional  military 
considerations).  With  this  in  mind,  the  current  efforts  have  focused  on  the  Operational 
Assessment  Team  (OAT)  residing  in  the  Strategy  Division.  Little  attention  to  date  has 
been  focused  on  this  team  and  the  important  feedback  that  they  provide  in  the  overall 
EBO  process.  Our  effort  will  provide  this  team  with  critically  needed  visualization 
technology  to  support  analysis  and  decision  processes  that  address  the  challenges 
outlined  above. 

The  activity  of  the  OAT  in  an  effects-based  environment  focuses  on  addressing 
several  critical  questions: 
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Are  we  achieving  the  effects  desired  by  the  Joint  Forces  Air  Component 
Commander  (JFACC)? 

What  is  our  confidence  that  we  are  achieving  the  effects? 

Does  the  intended  effect  have  the  desired  duration;  if  not,  how  long  and  we  expect 
the  effect  to  persist? 

Are  there  any  unintended  effects;  if  so,  what  implications  do  these  have  for  the 
plan? 

Do  we  need  to  modify  our  plan? 

Do  we  need  to  modify  our  assessment  strategy? 

These  are  a  few  of  the  current  challenging  questions  that  a  new  technology  has  to  support 
with  decision  aiding  tools  for  the  OAT.  In  addition  to  answering  these  questions,  Effects- 
based  Assessment  (EB  A)  considers  not  just  the  direct  effects  of  the  attack,  but  also 
indirect  effects  in  multiple  domains,  and  the  mechanisms  that  link  the  effects  between 
those  domains,  systems  and  targets  (Waller,  2003).  When  taking  such  a  complex  view  to 
perform  assessment,  the  tools  designed  to  support  this  process  have  to  be  developed 
around  a  deep  and  rich  understanding  of  the  cognitive  functions  an  operator  uses  to  form 
judgments,  carry  out  analysis  and  make  decisions.  Lastly  there  is  a  need  to  understand 
the  expert  novice  differences.  This  will  aid  in  understanding  any  workarounds  performed 
and  other  strategies  developed  to  perform  assessment. 

In  the  remainder  of  this  paper  we  will  discuss  our  integrated  cognitive  and  systems 
engineering  approach  to  the  development  of  visualization  technologies  designed  to 
support  effects-based  operational  assessment.  We  first  will  describe  our  integrated 
cognitive  and  systems  engineering  approach.  We  follow  this  with  a  description  of  our 
visualization  interface  concept  development.  We  end  with  a  brief  description  of  the 
evaluation  of  the  concepts. 


Cognitive  Systems  Engineering  for  Operational  Effects  Assessment 

Our  cognitive  systems  engineering  (CSE)  of  the  EBA  domain  emphasized  five 
focal  areas  or  views:  Work  management,  cognitive  work,  products,  collaboration  and 
automation.  The  concepts  comprising  these  views,  along  with  the  relationships  between 
them,  are  shown  in  Figure  1.  Each  view  is  presented  in  Figure  1  as  colored  concepts 
related  by  one  or  more  bi-directional  relationships.  All  of  the  CSE  analysis  that  we 
carried  out  conformed  to  this  informal  top-level  ontology.  While  all  five  views  are 
represented  in  our  analysis,  we  concentrated  the  majority  of  our  efforts  on  understanding 
the  cognitive  work  requirements  associated  with  EBA.  It  is  the  cognitive  work  associated 
with  the  job  of  assessment  that  drives  the  operational  and  functional  support  requirements 
for  those  activities  comprising  assessment.  The  concepts  and  relationships  surrounding 
this  cognitive  work  are  shown  to  the  right  of  Figure  1.  The  cognitive  work  required 
during  EBA  includes  workflow,  cognitive  work  requirements,  information  requirements 
that  directly  support  the  cognitive  work,  data  and  data  sources  directly  supporting  the 
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information  requirements,  and  constraints  and  dependencies  that  influence  the 
organization  of  work  flow. 


aids-in/ 
augmented-by 


, - -= — £ - .  i  i  \ 

I  participants  1  responsible-for/  use/  create/ 
v  J  carried-out-  by  used-by  created-b 


supports/ 
supported-by  * 


Figure  1:  CSE  Ontology  for  Analysis  of  the  EBA  Domain 


Working  with  Air  Force  documents  and  subject-matter  experts  (SMEs)  we 
defined  15  high-level  functions  representing  the  cognitive  work  associated  with  EBA  (see 
Table  1).  These  were  then  decomposed  into  a  series  of  activities  representing  the 
structure  of  each  function.  With  operational  activities  as  the  centerpieces  of  each 
diagram,  we  then  defined  the  work  management  strategies,  products,  collaboration  and 
automation  associated  with  each  activity.  In  general,  the  work  associated  with 
operational  activities  is  expressed  as  a  relatively  small  set  of  perceptual  and  cognitive 
processes.  Table  2  contains  a  listing  of  these  processes.  The  method  we  used  in 
constructing  the  concept  maps  required  that  each  element  of  perceptual  or  cognitive  work 
be  stated  in  terms  of  these  primitives.  This  is  important  for  several  reasons:  (1)  it  allowed 
us  to  express  the  low-level  work  involved  in  EBA  as  a  small  set  of  primitives,  thereby 
allowing  us  to  better  manage  the  move  to  system  requirements  in  later  stages  of  the 
design,  (2)  it  provided  a  way  to  map  our  system  design  requirements  to  results  in  the 
human  factors  research  literature  and  (3)  it  provided  a  bridge  between  the  cognitive 
systems  analysis  and  the  system  engineering  model  of  EBA  to  be  discussed  below. 
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Table  1.  EBA  High-level  Cognitive  and  Perceptual  Work 


Pre-execution 

Execution 

Post-execution 

Assessment  planning 

Accrue  evidence 

Provide  inter-division 
feedback  on  operational 
effectiveness 

Determine  adversary 
capabilities  and  likely  COA 

Analyze  operational  results 

Develop  JAOP 

Integrate  BDA 

Develop  STTM 

Execution  tracking 

OAT  management  of  EBA 

Integrate  functional  damage 
assessment 

Predict  operational 
effectiveness 

Integrate  mission 
assessment 

Integrate  physical  damage 
assessment 

Integrate  target  system 
assessment 

Table  2.  EBA  Cognitive  and  Perceptual  Work  Elements 


Comparison 

Interpret 

Inference 

Integrate 

Recognition 

Derive 

Decision 

Extrapolate 

Selection 

Predict 

Search 

Correlate/  associ  ate 

Monitor 

Evaluate 

Acquire 

Identify 

Communicate 

Rank 

Assess 

In  the  discussion  that  follows,  we  concentrate  on  activities  comprising  the 
execution  portion  of  EBA.  These  include  (1)  accruing  input  from  lower-level  processes 
and  from  other  data  sources,  (2)  fusing  the  data,  (3)  analyzing  the  data,  (4)  determining  if 
objectives  were  being  met,  (5)  troubleshooting  the  plan  (or  the  assessment  indicators)  if 
necessary,  and  (6)  organizing  assessment  findings  for  dissemination  to  the  IF  ACC  and/or 
the  strategy  team. 

Inspecting  the  categories  of  work  appearing  in  each  operational  activity  is 
informative  in  helping  us  understand  the  technology  support  requirements  for  the 
activities.  Accruing  evidence  is  primarily  a  process  of  acquisition  and  interpretation  or 
assessment.  Most  of  the  acquisition  processes  defining  the  work  of  this  activity  included 
either  actively  seeking  desired  information  or  monitoring  pre-determined  information 
channels  for  useful  information  that  could  be  acquired  opportunistically.  As  information 
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was  obtained  it  was  assessed  or  interpreted  in  terms  of  the  plan  in  force  for  a  particular 
mission. 

Analyzing  operational  results,  though  containing  many  work  elements  ranging 
across  16  different  perceptual  and  cognitive  categories,  consists  almost  totally  of 
interpretation  and  assessment  work.  This  seems  reasonable,  given  that  success  in  this 
operational  activity  must  relies  on  combining  large  amounts  of  information  from  many 
sources  into  a  coherent  “story”  and  then  trying  to  understand  if  the  “story”  is  the  right  one 
within  the  context  of  the  current  objectives,  tasks,  indicators  and  timelines.  The 
remaining  perceptual  and  cognitive  work  for  this  activity  exist  only  to  support  these  other 
two  primary  functions. 

We  developed  a  hybrid  concept  map,  which  we  termed  “integrate  mission 
assessment,”  that  captured  elements  of  both  determining  mission  effectiveness  and 
troubleshooting  the  effects-based  plan  when  events  demonstrated  plan  shortfall.  From  a 
cognitive  systems  engineering  point  of  view  this  was  an  interesting  activity.  Whereas 
determining  mission  effectiveness  involved  a  single  process,  “evaluate,”  troubleshooting 
the  plan  was  a  more  heterogeneous  process  that  included  evaluating  available 
information,  making  inferences  about  what  went  wrong,  and  making  specific  decisions 
about  how  to  repair  the  plan. 

In  all,  we  found  that  all  of  the  perceptual  and  cognitive  work  comprising  the 
execution  portion  of  EBA  could  be  accounted  for  with  19  functional  terms,  as  shown  in 
Table  2.  This  is  a  slightly  higher  number  of  terms  than  was  identified  by  Hale  (1988)  but 
significantly  fewer  than  identified  by  Fineberg  (1995).  We  are  now  working  to  determine 
the  system  design  implications  of  each  term. 

The  EBA  concept  maps  then  were  mapped  to  system  engineering  requirements. 
This  was  an  important  step  in  ensuring  that  the  CSE  analysis  results  were  represented  in 
the  overall  system  development.  We  accomplished  this  by  mapping  items  within  five 
cognitive  system  engineering  categories  to  elements  of  traditional  functional  analysis. 
System  engineering  requirements  analysis  products  then  were  developed  from  the 
functional  analysis.  These  mappings  are  shown  in  Figure  2.  The  five  CSE  categories 
mapped  information  about  the  work  domain,  control  tasks  performed  by  assessors, 
strategies  carried  out  in  support  of  requirements,  socio-organizational  factors  and  human 
limitations  and  competencies.  The  mappings  from  CSE  to  functional  categories  were 
many-to-many,  as  would  be  expected  for  each  of  these  broad  CSE  categories.  The 
system  engineering  functional  analysis,  which  now  included  the  information  from  the 
cognitive  system  engineering  analysis,  formed  the  basis  for  development  of  system 
engineering  requirements  analysis  products.  These  fell  into  three  broad  categories  or 
views:  Operational  views,  functional  views  and  physical  views. 
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Work  Domain 
Constraints 
Dependencies 
Opportunities 
Afford  ances 
Requirements 
Goals 

Physical  Form 


Control  Tasks 

Required  Activities 
Inputs  and  outputs 
Knowledge  States 
Shunts 


Systems  Engineering 
Requirements  Analysts  Activities 


Strategies 
Strategies  to 
execute  activities 
Specific  in/oul  puts 
Triggering  events 
Critical  decisions 


Sodo-Organbationa  (_ 

•  Org  Structures 

•  Conditions  triggering  | 
Org  struct  changes 

■  Who  does  what 
when 

■  Coord  dynamics  at 
team  and  org  levels 

1  Coflaborational 
dynamics  at  team 
and  org  levels 

•  Ad-hoc  teams 
triggering  events 

■  Factors  triggering 
setf-org  behavior 


Systems  Engineering 
Requirements  Analysis 
Products 


Operational  View  of 
System 

1  OP  Need  Definition 
OP  Sequences 
OP  Environments 
Stimulating  Cond/ 
Events 

OP  Constraints 
Mission 

Performance  Reqs 
User  rotes  anti  reqs 
OP  Interlaces 


Functional  View  of 

SYS  Functions 

SYS  Performance 

Tasks  or  Actions 

Performed 

Interfunction 

Relationships 

HW/-SW 


Performance 
Constraints 
Verification  Reqs 


Physical  View  of 
Syatem 
System 
Configuration 
Characterization  of 
System  Users 
System  Physical 
Limitations 


Figure  2.  Cognitive  Systems  to  Systems  Engineering  Requirements  Mapping 

All  of  the  information  from  the  previous  analyses;  along  with  EBA  doctrine, 
concepts  and  other  documentation,  and  SME  input;  was  used  to  develop  a  set  of  system 
engineering  models  of  EBA  in  CORE.  CORE  is  a  commercially  available  CASE  tool 
that  allows  management  of  an  entire  development  project.  Modelers  can  include  a  wide 
range  of  information  about  system  functionality,  inputs  and  outputs,  sequence,  constraints 
and  dependencies,  triggering  conditions  and  end-states,  products  and  so  on.  The  system 
also  includes  a  discrete  event  simulation  engine  that  enables  exploration  of  tradeoffs  and 
what-if  analyses.  Output  can  be  in  terms  of  any  standard  DoDAF  formalism. 

Our  system  engineering  model  consists  of  108  functions  focused  primarily  on  the 
execution  portion  of  EBA.  The  functions  are  organized  hierarchically,  as  shown  (in  part) 
in  Figure  3.  Referring  to  this  figure,  each  component  function  is  shown  as  a  block  within 
its  parent  diagram.  Functional  boundaries  for  a  particular  activity  are  shown  with  block 
shading;  light  blocks  are  within  a  boundary  while  shaded  blocks  are  outside  a  boundary. 
Sequence  and  flow  logic  also  are  shown  in  each  diagram.  Most  of  the  detailed  cognitive 
and  perceptual  work  across  the  five  top-level  diagrams  took  place  in  support  of  analyzing 
operational  results.  This  can  be  seen  by  considering  the  number  of  levels  of 
decomposition  needed  to  completely  describe  each  function.  A  complete  description  of 
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analyzing  operational  results  required  decomposition  to  eight  levels,  while  adequate 
descriptions  of  the  remaining  four  functions  within  execution  required  no  more  than  3 
levels  of  decomposition.  In  order  to  preserve  the  information  contained  in  the  concept 
maps,  as  well  as  the  mappings  between  concept  map  data  and  system  engineering 
requirements,  we  placed  an  emphasis  in  CORE  model  development  on  capturing  the 
cognitive  and  perceptual  work  within  the  functional  system  description.  Thus,  cognitive 
and  perceptual  work  appears  as  separate  functions  in  the  lower  levels  of  the  CORE 
diagrams.  When  combined  with  their  attendant  inputs  and  outputs  these  work  elements 
generate  technology  development  and  visualization  requirements  directly. 


Figure  3.  A  Portion  of  a  System  Engineering  Model  for  Operational  Assessment 

The  completed  CORE  diagrams  formed  the  basis  of  requirements  identification. 
The  CORE  package  automatically  generates  system  requirements.  Using  this  facility  we 
generated  a  requirements  set  consisting  of  330  items.  The  development  team  then  edited 
this  set  to  determine  what  items  fell  within  the  scope  of  the  development  program. 
Because  the  cognitive  and  perceptual  work  from  the  concept  mapping  analysis  was 
distributed  across  the  system  engineering  model  hierarchy,  the  requirements  associated 
with  this  work  also  appeared  hierarchically. 

Visualization  Interface  Concept  Development 

Based  on  our  in-depth  analysis,  the  fundamental  process  of  assessment  was 
broken  down  into  five  categories:  Request  for  Information  (RFI),  Evidence  Accrual, 
Analyze  Operational  Results,  Determine  Effectiveness  of  Air  Operations  in  Achieving 
Objectives,  and  Develop  Assessment  Briefing.  These  five  high  level  categories 
encompass  the  processes,  tasks,  data,  and  information  needed  to  perform  operational 
assessment  during  execution.  By  breaking  assessment  into  these  categories,  we  were 
able  to  fully  analyze  the  data.  This  in  turn  helped  us  to  ensure  that  the  tasks  and  decision 
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making  skills  were  supported  in  the  design  of  the  concepts.  Below,  we  will  go  into  more 
detail  about  the  categories  and  discuss  initial  concept  ideas. 

Request  for  Information 

There  are  several  tools  that  support  the  RFI  generation  process.  However,  many  of 
these  either  are  not  being  used  or  have  not  been  deployed.  It  will  be  important  to  provide 
and  deploy  a  tool  that  makes  this  task  consistent  and  easy  to  carry  out.  If  such  a  tool  is 
available,  the  process  may  then  be  standardized.  In  addition  to  having  standardized 
processes  in  place  it  should  also  be  possible  to  save  the  RFIs  themselves  in  an  archive. 
This  makes  them  available  for  possible  future  use,  thereby  improving  the  efficiency  of 
assessors  over  time.  Functionality  that  should  be  included  in  a  RFI  support  module  are: 

•  What  is  being  requested. 

•  Who  is  being  asked  for  the  information. 

•  The  type  of  data  needed. 

•  Data  format. 

•  Data  request  deadline. 

•  Any  additional  comments  on  specifics  of  the  data  needed. 

Evidence  Accrual 

Once  data  have  been  requested,  the  assessor  then  needs  to  manage  the  data  in  a 
meaningful  way.  This  implies  storing  the  data  and  naming  the  file  in  a  convention  that 
the  user  will  understand  and  that  will  facilitate  locating  the  file  rapidly.  Users  must  be 
able  to  allocate  the  data  to  a  particular  day,  objective  and/or  action.  This  allows  the 
assessor  to  view  at  a  glance  which  objectives/actions  have  data  collected  against  them, 
giving  them  an  indication  what  data  still  needs  to  be  collected.  However,  this  high  level 
glance  is  not  enough  to  tell  the  assessor  anything  about  the  confidence  in  that  data.  This 
shortfall  raises  the  need  for  a  confidence-based  rating  system.  Confidence  ratings  should 
include  source,  reliability,  and  validity.  These  attributes  are  important  because  different 
data  types  and  their  original  sources  give  assessors  an  indication  of  the  true  state  of  the 
event  taking  place.  Some  sources  are  more  reliable  than  others,  however,  and  in  many 
cases  data  will  be  in  conflict.  When  conflicting  data  are  gathered,  assessors  will  naturally 
rely  on  the  data  that  they  believe  is  more  reliable.  Their  beliefs  might,  or  might  not  be, 
correct.  Thus,  the  visualization  system  should  provide  cues  about  data  source,  reliability, 
validity  and  the  age  of  the  data. 

Analyze  Operational  Results 

As  the  operation  proceeds  and  data  are  streaming  in,  assessors  need  to  evaluate 
that  data  against  the  plan  of  the  day  to  estimate  progress  toward  intended  effects.  The 
plan  is  based  on  actions  that  need  to  be  taken  to  successfully  achieve  an  Operational 
Objective  and/or  an  Effect.  One  critical  issue  associated  with  this  process  is  that  data 
often  are  delayed,  noisy,  or  missing  entirely.  The  assessors  must  make  their  estimates  in 
the  face  of  this  uncertainty.  Much  of  the  available  data  will  be  presented  in  the  form  of 
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summarized  tactical  results;  including  Battle  Damage  Assessment  (BDA),  Mission 
Effects  Assessment  (MEA),  Mission  Assessment  (MA);  and  intelligence  data.  From  this 
information  the  assessor  will  compare  actual  versus  planned  actions.  This  allows  the 
assessor  to  determine  if  the  plan,  in  terms  of  actions  per  time  period,  is  proceeding  as 
intended  or  not.  This  in  turn  allows  the  assessor  to  make  weight  of  effort  and  other 
recommendations  for  upcoming  days. 

Once  tactical  results  summaries  have  been  obtained  and  analyzed  assessors  must 
determine  if  the  intended  effects  are  being  achieved.  The  complexity  of  the  battlespace, 
combined  with  multidimensional  considerations  and  the  use  of  joint  and  coalition  forces, 
has  vastly  increased  the  complexity  of  these  estimates  (Diehl  &  Sloan,  2005).  The 
challenge  is  further  heightened  by  the  need  to  evaluate  unintended  and  nth-order  effects, 
and  to  convert  qualitative  evaluations  into  quantitative  estimates.  Current 
implementations  of  technology  intended  to  address  these  challenges  rely  on  attrition- 
based  metrics.  For  example,  one  approach  is  to  use  Desired  Mean  Point  of  Impact 
(DMPI)  as  a  count  to  assess  whether  the  effect  is  being  achieved.  However,  this  method 
fails  to  support  the  kind  of  assessments  required  to  relate  actions  to  intended,  unintended 
and  nth  order  operational  effects. 

In  order  to  analyze  the  data  to  see  if  an  effect  is  being  achieved  the  assessor  needs 
to  understand  the  battlespace  and  have  data  that  pertains  to  the  actions  and/or  tasks  that 
were  intended  to  cause  an  effect  to  occur.  Once  these  data  are  acquired,  the  assessor  can 
then  find  indications  of  the  effect  being  achieved  by  evaluating  the  adversary’s  actions 
and  systems.  In  addition,  assessors  can  plot  the  actions  and  their  associated  effects  on  a 
map.  The  data  then  can  provide  insight  into  any  unintended,  2nd  order,  and/or  3rd  order 
effects. 

Determine  Effectiveness  of  Air  Operations  in  Achieving  Objectives 

The  difference  between  this  category  and  Analyzing  Operational  Results  is  the 
fact  that  the  assessor  is  now  comparing  their  analysis  of  their  operational  results  to  the 
overall  Operational  Objective.  The  overarching  goal  of  an  operation  is  to  successfully 
complete  and  achieve  an  Effect  and  the  Operational  Objectives  in  the  time  allotted. 

Develop  Assessment  Briefing 

The  OAT  chief  briefs  the  JFACC  twice  daily  (morning  and  evening).  The 
preparation  for  these  briefings  is  often  time  consuming,  diverting  resources  away  from 
their  primary  job  of  assessment.  The  issues  surrounding  preparation  of  assessment 
briefings  are  similar  to  those  of  RFI  generation  and  management:  There  currently  are  no 
standardized  tools  that  have  been  deployed  to  aid  in  this  task.  The  fundamental  goal  of 
our  system  will  be  to  help  assessors  reduce  the  time  spent  in  developing  these  briefings. 
The  first  design  task  is  to  standardize  the  briefing.  This  standardized  format  would 
include  an  explanation  of  status  in  terms  of  actions  and  effects,  a  predictive  analysis,  and 
recommendations.  Based  on  this,  the  ideal  would  be  to  directly  take  a  snapshot  of  the 
current  status  of  an  Objective  and  place  that  into  the  briefing  in  summary  form.  If  this 
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summary  needs  to  be  modified  assessors  should  be  able  to  do  so  within  the  briefing 
through  links  to  primary  data  held  elsewhere  within  the  assessment  system.  Currently, 
this  concept  needs  further  analysis  to  determine  what  additional  data  might  be  needed, 
and  how  the  snapshot  can  be  transferred  seamlessly  into  a  briefing.  But,  as  stated  before, 
the  goal  is  to  reduce  the  time  spent  in  developing  these  presentations  so  that  the  assessor 
has  more  time  to  perform  other  tasks. 

Concept  Evaluation 

Throughout  our  concept  development  we  have  involved  several  subject  matter 
experts  (SME).  This  process  has  helped  us  to  ensure  that  1)  We  were  supporting  the 
tasks  that  need  to  be  supported,  2)  the  concepts  were  usable  and  made  sense,  and  3)  we 
understood  the  relationships  between  the  realities  of  the  domain  and  doctrine.  The 
process  from  data  collection  to  design  included  analyzing  the  data  we  collected  and 
developing  initial  concepts.  These  initial  concepts  were  then  evaluated  by  our  SMEs. 
Evaluation  consisted  of  working  through  each  of  the  assessment  tasks  using  the 
visualization  concepts  within  the  context  of  a  short  operational  scenario.  As  the 
evaluation  progressed  difficulties  in  procedure,  workflow,  data  access,  interpretation  of 
visualization  elements  and  any  comments  on  the  part  of  the  SMEs  were  documented  and 
categorized  with  respect  to  the  cognitive  work  elements  and  system  requirements.  We 
then  modified  the  concepts  based  on  these  data. 

This  evaluation  process  will  take  place  iteratively  until  the  “final”  concepts  have 
been  designed  and  are  ready  for  coding.  Once  coding  is  done,  the  prototype  will  be  taken 
to  a  major  Air  Force  experiment  in  April,  2006  to  be  evaluated  in  an  operational  setting. 

Conclusion 

Today’s  operations  are  exponentially  more  complex  than  before,  owing  to 
considerations  of  multiple,  interacting  domains;  a  need  to  assess  causal  linkages  across 
lines  of  effect;  a  need  to  assess  n^-order  effects;  and  the  requirement  to  “roll  up” 
operational  results  across  multiple  levels  of  effect.  New  technologies  being  developed 
will  need  to  enhance  operators’  situational  awareness,  help  them  effectively  manage 
uncertainty  and  risk,  and  aid  them  in  formulating  and  executing  decision  making 
strategies.  These  types  of  tools  are  critical  to  executing  a  successful  campaign.  During 
Operation  Iraqi  Freedom,  tools  such  as  these  were  not  available  to  the  OAT.  Without 
such  tools  the  OAT  found  it  difficult  to  evaluate  the  effectiveness  of  operations.  An 
effective  visualization  system  can  help  tremendously  with  these  challenges  by  supporting 
the  assessment  warfighter  with  procedures  and  processes  and  by  making  available  the 
data  required  to  accurately  assess  progress  toward  effects.  In  short,  helping  the  OAT  to 
understand  where  they  were,  where  they  are  and  where  they  will  be  informs  assessment, 
thereby  allowing  the  OAT  to  keep  the  entire  AOC  current  and  well-informed  as  to  the 
effectiveness  of  an  operation. 
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